Reducing Memory and Traffic Requirements for Scalable Directory-Based Cache Coherence Schemes

As multiprocessors are scaled beyond single bus systems, there is renewed interest in directory-based cache coherence schemes. These schemes rely on a directory to keep track of all processors caching a memory block. When a write to that block occurs, pointto-point invalidation messages are sent to keep the caches coherent. A straightforward way of recording the identities of processors caching a memory block is to use a bit vector per memory block, with one bit per processor. Unfortunately, when the main memory grows linearly with the number of processors, the total size of the directory memory grows as the square of the number of processors, which is prohibitive for large machines. To remedy this problem several schemes that use a limited number of pointers per directory entry have been suggested. These schemes often cause excessive invalidation traffic. In this paper, we propose two simple techniques that significantly reduce invalidation traffic and directory memory requirements. First, we present the coarse vector as a novel way of keeping directory state information. This scheme uses as little memory as other limited pointer schemes, but causes significantly less invalidation traffic. Second, we propose sparse directories, where one directory entry is associated with several memory blocks, as a technique for greatly reducing directory memory requirements. The paper presents an evaluation of the proposed techniques in the context of the Stanford DASH multiprocessor architecture. Results indicate that sparse directories coupled with coarse vectors can save one to two orders of magnitude in storage, with only a slight degradation in performance.